Hybrid query execution plan

ABSTRACT

A procedural pattern in a received query execution plan can be matched to a stored pattern for which an equivalent declarative operator has been pre-defined. The query execution plan can describe a query for accessing data. A hybrid execution plan can be generated by replacing the procedural pattern with the equivalent declarative operator. A hybrid execution plan processing cost can be assigned to execution of the hybrid execution plan and a query execution plan processing cost can be assigned to execution of the query execution plan. The assigning can include evaluating a cost model for the hybrid execution plan and the query execution plan. The query can be executed using the hybrid execution plan if the hybrid execution plan processing cost is less than the query execution plan processing cost or the query execution plan if the hybrid execution plan processing cost is greater than the query execution plan processing cost. Related systems, methods, and articles of manufacture are disclosed.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. Provisional Patent ApplicationNo. 61/391,007, filed on Oct. 7, 2010 and entitled “Turning a ProceduralDomain-Specific-Query-Language (DSQL) Into a Hybrid Data-Flow Graph,”the disclosure of which is incorporated herein by reference in itsentirety.

TECHNICAL FIELD

The subject matter described herein relates generally to data processingand, in particular, to generating a hybrid data flow execution planbased on domain specific query language for obtaining and transformingdata from a database.

BACKGROUND

In software development or domain engineering, a domain-specificlanguage is a programming language or a specification language dedicatedto a particular problem domain, a particular problem representationtechnique, and/or a particular solution technique. A domain-specificlanguage is created to solve problems in a particular domain and mightnot be intended to solve problems outside it.

Many applications provide domain-specific-query-languages (“DSQL”) toallow users to express custom and often data-intensive business logic.Various traditional relational databases provide a limited set ofoptions for executing business logic within a database. Some of theseinclude various procedures and/or a series of complex queries coupledtogether with code. However, these methodologies are generally eitherpart of the application implementation and therefore unchangeable by theuser to implement custom functions or require the user to deal withgeneral stored procedure languages and database internals to developcustom business logic. Additional problems can arise if all businesslogic is expressed on application side. As a result, large amounts ofdata are transferred between an application and a database, which can betime consuming and can prevent users from experiencing a full potentialof a dataset.

Relational databases can be reliable and scalable as well as can bebased on SQL in order to provide a standardized and powerful querylanguage. However, some recent trends in the technology of relationaldatabase have been trying to differentiate themselves from classicrelational database management systems. One of these trends includesNoSQL database management systems which might not require fixed tableschemas, usually avoid join operations and typically scale horizontally.While the classic relational database model systems works well for mostenterprise applications, there exist applications where specific querylanguages can be provided to the user for easy interaction with the datastored in a database. Thus, to keep a user within confined boundaries ofan application domain, no SQL may be needed. The DSQL systems allowusers to develop data-intensive processing logic in a domain specificnon-SQL language that still benefits from execution within a database,thereby allowing access to the database.

SUMMARY

In one aspect, a computer-implemented method includes receiving a queryexecution plan describing a query for accessing data and including aprocedural pattern. The procedural pattern is matched to a storedpattern for which an equivalent declarative operator has beenpre-defined. A hybrid execution plan is generated by replacing theprocedural pattern with the equivalent declarative operator. A hybridexecution plan processing cost is assigned to execution of the hybridexecution plan and a query execution plan processing cost is assigned toexecution of the query execution plan. The assigning includes evaluatinga cost model for the hybrid execution plan and the query execution plan.The query is executed using the hybrid execution plan if the hybridexecution plan processing cost is less than the query execution planprocessing cost or the query execution plan if the hybrid execution planprocessing cost is greater than the query execution plan processingcost.

In some variations one or more of the following can optionally beincluded. A second hybrid execution plan can be generated by replacing adifferent procedural pattern with a second equivalent declarativeoperator. A second hybrid execution plan processing cost can be assignedto execution of the second hybrid execution plan by evaluating a costmodel for the second hybrid execution plan. The query can be executedusing the second hybrid execution plan if the second hybrid executionplan processing cost is less than both of the query execution planprocessing cost and the hybrid execution plan processing cost.

The matching can further include applying tuple calculus to identify theprocedural statement for replacement by the pre-defined equivalentdeclarative statement. Evaluating the cost model can include determiningthe hybrid execution plan processing cost and the query execution planprocessing cost using functions that include:

Cost=Σ_(k) ^(N) ^(O) ^(-N) ^(P) c _(decl)+Σ_(m) ^(Np) c _(m)^(proc)+Σ^(C) ^(parop) /min(P _(C) ,P _(p))

where Σ_(k) ^(N) ^(O) ^(-N) ^(p) c_(decl) represents a first sum ofcosts for all declarative statements, Σ_(m) ^(N) ^(p) c_(m) ^(proc)represents a second sum of costs for all procedural statements, andΣ^(c) ^(parop) /min(P_(C),P_(p)) represents a third sum of costs for allprocedural and/or declarative operators that are calculated in parallelas divided by a minimum number of available parallel processors (P_(C))and parallel operators (P_(P)). The equivalent declarative operator caninclude at least one of a selection declarative operator, a joindeclarative operator, a leftouter-join declarative operator, aprojection declarative operator, an aggregation declarative operator,and an assign declarative operator.

A second procedural pattern in the query plan can be translated to asecond equivalent declarative operator in the hybrid execution plan. Thetranslating can include identifying a borderline procedural patternassociated with a side effect when the borderline procedural pattern isexecuted. The side effect can enable a condition that when a first tupleelement value is changed during a loop iteration of the borderlineprocedural pattern, the changed tuple element value is accessed in asubsequent loop iteration such that access of the changed tuple elementvalue rather than access of the first tuple element value prevents theborderline procedural pattern to be translated to an equivalentdeclarative operator.

Articles are also described that comprise a tangibly embodiedmachine-readable medium operable to cause one or more machines (e.g.,computers, etc.) to result in operations described herein. Similarly,computer systems are also described that may include a processor and amemory coupled to the processor. The memory may include one or moreprograms that cause the processor to perform one or more of theoperations described herein.

The details of one or more variations of the subject matter describedherein are set forth in the accompanying drawings and the descriptionbelow. Other features and advantages of the subject matter describedherein will be apparent from the description and drawings, and from theclaims.

DESCRIPTION OF DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, show certain aspects of the subject matterdisclosed herein and, together with the description, help explain someof the principles associated with the disclosed implementations. In thedrawings,

FIG. 1 illustrates an exemplary system for defining and monitoringbusiness conduct, according to some implementations of the currentsubject matter;

FIG. 2a-k illustrate various exemplary query operators;

FIG. 3 illustrates differences between results of executing proceduralquery language and declarative query language;

FIG. 4 illustrates an exemplary execution of a data query plan;

FIG. 5 illustrates results of executing the data query plan shown inFIG. 4;

FIG. 6 illustrates the data query plan shown in FIG. 4 as expressed withsimple tuple calculus and annotated with matching declarative operators;and

FIG. 7 shows a flow chart illustrating features of a method.

When practical, similar reference numbers denote similar structures,features, or elements.

DETAILED DESCRIPTION

There is a need for creating a faster and more efficient way to accessdata stored in a database using custom business logic. Further, there isa need to allow for a faster and more efficient way of accessingdatabase-stored data using a domain specific query language (“DSQL”) bydeveloping and executing an optimal hybrid data flow plan.

To address these and potentially other deficiencies of currentlyavailable solutions, one or more implementations of the current subjectmatter provide methods, systems, articles or manufacture, and the likethat can, among other possible advantages, provide systems and methodsfor providing systems, methods, and computer program products foraccessing and processing database-stored data using queries. Someimplementations of the current subject matter can be configured totranslate a DSQL into a hybrid data-flow execution plan containingdeclarative operators mixed with procedural operators and to provideruntime comparisons for both types. Further, a general tuple calculusthat captures declarative and procedural semantics can be implemented todevise a cost model that can be used to guide the translation processinto an optimal hybrid plan.

In some implementations, the current subject matter can be configured tobe implemented in a system 100, as shown in FIG. 1. The system 100 caninclude a processor 110, a memory 120, a storage device 130, and aninput/output device 140. Each of the components 110, 120, 130 and 140can be interconnected using a system bus 150. The processor 110 can beconfigured to process instructions for execution within the system 100.In some implementations, the processor 110 can be a single-threadedprocessor. In alternate implementations, the processor 110 can be amulti-threaded processor. The processor 110 can be further configured toprocess instructions stored in the memory 120 or on the storage device130, including receiving or sending information through the input/outputdevice 140. The memory 120 can store information within the system 100.In some implementations, the memory 120 can be a computer-readablemedium. In alternate implementations, the memory 120 can be a volatilememory unit. In yet some implementations, the memory 120 can be anon-volatile memory unit. The storage device 130 can be capable ofproviding mass storage for the system 100. In some implementations, thestorage device 130 can be a computer-readable medium. In alternateimplementations, the storage device 130 can be a floppy disk device, ahard disk device, an optical disk device, a tape device, non-volatilesolid state memory, or any other type of storage device. Theinput/output device 140 can be configured to provide input/outputoperations for the system 100. In some implementations, the input/outputdevice 140 can include a keyboard and/or pointing device. In alternateimplementations, the input/output device 140 can include a display unitfor displaying graphical user interfaces.

Some implementations of the current subject matter are directed toprocessing of data queries based on a procedural description in a moreefficient way by devising a query execution plan that combinesprocedural query language and declarative query language that caninclude various regular expressions, logic programming, and/orfunctional programming. Using a combination of procedural anddeclarative query languages, the current subject matter can beconfigured to allow a database layer to provide infrastructure to handlecomplex data processing tasks and move such tasks from an applicationlayer to the database layer and hence, closer to the actual datarequested in a query. This reduces costs associated with transferringlarge quantities of data between the application and database layers. Insome implementations, all data processing logic can be executed in thedatabase layer and by a “lightweight” (one that does not requiresubstantial processing or an increase in cost) application thatcoordinates the tasks and displays results of the query to a user. Someimplementations of the current subject matter can be used in businessplanning applications for implementing custom planning functions.

One way to implement complex data processing queries directly within adatabase layer can involve use of various stored procedures. However,such procedures can be targeted towards application developers or systemadministrators, whereas use of the DSQL-based queries can allow anapplication user to implement its custom application logic duringapplication runtime. To accomplish this, some implementations of thecurrent subject matter can be configured to translate a procedural DSQLquery into a declarative execution model of the underlying database inorder to create a hybrid query execution plan that can incorporatedeclarative and procedural logic into a single plan. At least someprocedural logic involved in the plan can be expressed in an entirelydeclarative way. Based on this plan, a cost model can be derived thatcan be used to perform requisite translations of DSQL query in order todevelop an optimal hybrid query execution plan. The cost model cantarget an in-memory system and thereby, reduce operator runtime costs.Further, the cost model can operate on a level of a logical executionplan and thus, does not incorporate various low level features,including but not limited to, cache sizes and cache hierarchies.

The following provides an illustrative discussion of a generalprocedural tuple calculus concept that can be used to generate a hybridquery execution plan, according to some implementations of the currentsubject. A general procedural tuple calculus can serve as the frameworkto express data driven procedural query scripts. It can allow fortranslation between procedural and declarative parts of the query scriptand also can enable building a unified cost model.

A basic entity of this calculus is a tuple, t. Each element of the tuplecan be accessed by position i as in t[i] or by a unique name thatreferences a certain tuple position −t[name]. Using assignment operator“:=”, a new value can be assigned to a tuple component. On a right-handside of an assignment, an arbitrary arithmetic expression of tuplecomponents or constants can be allowed. A number of components in atuple can be denoted as N. If a tuple is assigned a component positionlarger than its current size N or an unknown position name, then thetuple can be extended with a new value of component N+1, which can bereferred to by a new position name. Multiple tuples can form list(s) T.A foreach statement (i.e., a statement that traverses data in acollection of data) can iterate over each tuple in such list(s). Newtuple lists can be constructed using tuplelist( ) function. A statementadd(t, T) can add tuple t to a tuple list T. Further, to allow forconditional control flow, if conditions can evaluate a predicate pred onone or more tuples. A tuple comparison function cmp(t_(i), t_(j),c_(cmp)) can compare tuple t_(i) and t_(j) for each component given intuple c_(cmp). New tuples can be constructed using tuple( ). A functionlookup(t_(key), T) can find a tuple in a tuple list such thatcmp(t_(key), t, c_(cmp))=true and c_(cmp) can contain all componentposition names of the key tuple. A while(pred(t)) loop can iterate aslong as the predicate pred(t) is true.

Some implantations of the current subject matter can be configured totranslate a data-driven procedural DSQL query into a semanticallyequivalent hybrid execution plan. The hybrid plan can provide a set ofcommon declarative operators like projection, selection, join andaggregation (each of which is discussed below and illustrated inconnection with FIGS. 2a-2k ). These operators can be optimized toprocess even large input datasets efficiently. Thus, it is desirable toexpress procedural logic that can operate on large datasets in terms ofthe declarative operators.

FIGS. 2a and 2b illustrate use of an equivalent of a declarativeselection operator expressed with the tuple calculus for a simpleselection and selection with lookup, respectively. The selectionoperator can iterate over each tuple in its input and check if itsselection predicate is true for the current tuple. Given a list oftuples T and a predicate pred it can be written as shown in FIG. 2a . Aspecial form of predicate can check for certain components i of a tupleu to determine if they are equal to some constant value c or equal tothe component values j of the current tuple t within a loop. This formof selection can be expressed using a lookup function, as shown in FIG.2 b.

FIGS. 2c and 2d illustrate use of an equivalent of a declarativeselection operator expressed with the tuple calculus with a join and ajoin, respectively. The join in combination with a selection is similarto the selection with lookup shown in FIG. 2b . Every join can be anested loop over two tuple lists. One of the simplest forms of the join(in terms of a procedural description) is a Cartesian product. An outerloop of the join can iterate over all tuples tin a tuple list T and aninner loop can iterate over each tuple u in a list U. If the joininvolves a number of join attributes described by tuple j, then thedescription can be extended with a condition in the inner loop thatcompares the join components defined by tuple j for each combination oftuples t and u, as shown in FIG. 2d . In case of a Leftouter-Join, anempty tuple u can be created since no tuple u compares to t, as shown inFIG. 2 e.

FIG. 2f illustrates an exemplary projection operator expressed usingtuple calculus. The projection operator can project to all components pof a tuple and eliminate duplicates. To handle duplicates, a list P canbe used in combination with a lookup function. If the current tuple t isnot found in P, it can be added to P.

FIG. 2g illustrates an exemplary aggregation operator expressed usingtuple calculus. The aggregation operator can be used when an element mof a tuple t is accumulated over a set of tuples T using tuple u tostore the current sum during each iteration.

FIG. 2h illustrates an exemplary assign operator, which can be a specialcase of the projection operator. In this case, instead of restrictingcomponents of a tuple, the assign operator can either redefine anexisting component or add a new one. For a tuple t, either a componentt[i]=exp or a new component t[N+1]=exp can be assigned the value ofexpression exp where N is a number of components in the current tuple.

In some implementations, the current subject matter can be configured totranslate each of the procedural operators discussed above along withother parts of a DSQL script into their equivalent declarative form.Further, where translation might not be possible, a combination of theprocedural and declarative language can be used to create a hybrid queryexecution plan. The following discussion illustrates proceduralborderline scenarios, where translation of the procedural language intodeclarative language might not be clearly possible because uses ofprocedural language and declarative language may produce differentresults.

Although, most procedural constructs can match patterns for declarativeoperators and can be interchangeable, even subtle changes in theconstructs of such statements can break this equality. When procedurallogic is expressed using a declarative form, loops can be unrolledimplicitly and expressions can be calculated independently for eachtuple. As such, no side effects (i.e., during one loop iteration, atuple element value can be changed and a new value can be accessed in asubsequent loop iteration) can be allowed. Such side effects most oftenoccur when a tuple is used to transport state between calculationsduring different loop iterations.

FIG. 3 illustrates differences between uses of procedural anddeclarative languages. In the procedural case, the iterations can beexecuted sequentially and the second iteration can access the results ofthe first one. In the declarative case, the loop can be unrolled andmaterialized with a cross-join before the calculation can take place oneach row independently. Thus, each change to a tuple u in the formulacan affect different versions of the tuple u. FIG. 2i illustrates anexemplary procedural operator that is not side effect free. Thisoperator can be characterized by the fact that a tuple v is definedoutside of a loop, is part of an expression, and is also on theleft-hand side of an assignment inside the loop after it has beenaccessed in the expression.

FIG. 2j illustrates an exemplary procedural aggregation operator havinga side effect. The operators shown in FIGS. 2g and 2j can be similar,but the operator shown in FIG. 2j can introduce a tuple outside of theloop and the tuple that is used inside the loop as part of an expressionand as the left hand side of an assignment. Thus, the aggregation can bea borderline case of a procedural pattern with a side effect and onlybecause the declarative aggregation operator can internally use amechanism to carry state information (i.e., an aggregated sum) betweenmultiple iterations. As such, the aggregation operator can be used asdeclarative equivalent in that special circumstance. However, in someimplementations, substitutions of operations with this operator (e.g.,multiplication for a sum) can prevent translation into declarativelanguage.

FIG. 2k illustrates an exemplary while loop that cannot be translatedinto declarative language. The while loop can express semantics likeproduce M new tuples, which can conflict with semantics of a data-flowdeclarative plan, where an output of an operator can be determined bythe inputs it can receive. An equivalent declarative operator without aninput can be a leaf node in a graph, where leaf-nodes can only allowdata sources like tables and cubes. The constructs that preventtranslation into declarative language of the operator shown in FIG. 2kinclude a variable used inside the loop to carry state (i.e., it is notside effect free), and a number of iterations of the inner loop thatdepends on the component value of a tuple (i.e., it can be determined atruntime only).

The following illustrates an exemplary implementation of the currentsubject system that uses a combination of procedural and declarativestatements to develop a hybrid query execution plan based on the tuplecalculus concepts discussed above. The DSQL that is used in this examplecan be from an area of business planning and be part of an integratedbusiness planning functionality. FIG. 4 illustrates an exemplary scriptfor a business logic that can calculate a rolling plan, which can becreated multiple times a year as part of business planning (e.g., salesvalues for a company are planned at the beginning of a year for the next12 months). As the year proceeds and actual sales results arrive, theplanned values can be refined for the remaining months based on a newknowledge of actual sales figures. If the results so far are below theplan, the target for the remaining months can be raised to keep trackwith the overall goal of the year. If the results are better thanplanned, the target can be lowered. FIG. 5 illustrates results of therolling plan that include actual values provided for months of January,February, March, and April. Projected values (Version 1(before)) areindicated for months of May through December. Based on the received datafor January through April, expected values (Version 2(after)) areadjusted accordingly.

FIG. 4 illustrates how the rolling plan's DSQL can be expressed usingthe tuple calculus concept discussed above. Using the DSQL, it can bepossible to modify and/or create new data within the underlyingmultidimensional data model, i.e., a cube. A script can reference a cellin the cube through tuples written in curly brackets (e.g., {measure,dim₁, . . . , dim_(N)}. The first value can address the measure. Foreach of the cube's dimensions, a dimension value can be specified. Insome implementations, formulas within a script can reference only acertain sub-cube and thus, most of the dimensions can be constant forevery tuple within the sub-cube. The division of the cube in fixeddimensions and variable dimensions can relieve the user from specifyingdimension values for each dimension of the cube. The fixed dimensionscan describe the sub-cubes for which the script can be executed and onlythe variable dimensions can be necessary to describe a tuple. This meansthat for each script, there is an implicit loop over all possiblesub-cubes as defined by the fixed dimensions. Referring to FIG. 4, thefixed dimensions are a customer and a product. Thus, each distinctcombination of a customer and a product can define a sub-cube for whichthe rolling plan script can be calculated. Finally, the type of avariable can be defined in terms of dimension names, which means that avariable can take values from the domain of the dimension it refers toand can share the same type.

In FIG. 4, the first loop can be an iteration over all distinctcombinations of the fixed dimensions. All logic of the actual script canbe executed in the context of this loop. The second loop corresponds tothe first FOREACH statement in the example script. It loops over alldistinct values of the fiscal period dimension within the currentsub-cube. Within the loop the IF statement and the assignment tovariable SUM are expressed. For the calculation of the DELTA value, twolookups can be performed. Then, the old value in version 0 isoverwritten with the value of version 1. Finally, there is anotheriteration over all fiscal period values, again with the IF statementinside and the assignment and calculation of the new plan value for thecurrent fiscal period. The complete script written with the introducedtuple calculus is shown in FIG. 6. Every pattern that matches adeclarative operator is annotated in this figure. As can be seen, thewhole script can be completely translated into declarative operators.

In some implementations, an exemplary cost model can be generated usingthe declarative query language to determine effectiveness of a hybridquery execution plan or a plan that contains purely declarative querylanguage. The cost model can depend on the size of input data andtype(s) that are accessed during execution of the plan. It should benoted that for small sizes of data, a single procedural node capturingthe complete custom logic can be most efficient. However, whencardinalities (i.e., sizes of data) grow, plans with declarativeoperators can be more beneficial, because they can be betterparallelized and the operators can be optimized to handle largerdatasets more efficiently. Between the small size data and large sizedata, hybrid plans using declarative and procedural operators discussedabove can also be beneficial, as long as the separation of the remainingprocedural code parts does not lead to procedural operators with only asmall portion of code. The cost model discussed below can be configuredto weigh these different variants against each other and derive anoptimal hybrid execution plan that can handle complex scripts on largedata sets.

The cost model can be configured to consider various dimensions that caninfluence execution cost of such hybrid plan. First, the cost can dependon the size N_(D) of the input data set. Further, the cost can depend ona structure of the hybrid plan. This means that the separation ofprocedural parts into multiple procedural operators can be penalizedwith some overhead cost for each procedural operator N_(P). N_(O) cancorrespond to the number of all operators, such that the number ofdeclarative operators is N_(O)-N_(P). Multiple nodes (e.g., declarativeand/or procedural) in the hybrid plan that do not depend on each other,can be executed in parallel. The cost can also depend on a degree ofparallelism P_(C) in terms of available processors and a degree ofparallel operators in the resulting hybrid plan P_(P) (i.e., processorsand/or operators that can perform various tasks in parallel,respectively). A cost function can be defined as follows, where theresulting cost measure is a real number:

C _(plan)(S,N _(D) ,N _(O) ,N _(P) ,P _(C) ,P _(O))→R  (1)

Table 1 summarizes the input parameters.

TABLE 1 Input Parameters. Parameter Description N_(D) Size of input dataN_(O) Number of operators N_(P) Number of procedural operators P_(C)Number of processors P_(O) Number of parallel operators

While the input parameters N_(D) and P_(C) cannot be influenced by thetranslation process, the parameters N_(P) and P_(P) are subject tochange depending on the resulting hybrid plan. To measure the costs of aplan based on the tuple calculus each statement or statement block, incase of nested statements, is assigned a cost. In the simplest case ofserial execution within one procedural operator the cost is the sum ofcosts of all statement blocks c_(stmt)(N_(D)).

Cplan(S,N _(D) ,N _(O) ,P _(C) ,P _(p))=Σ_(s) ^(S)(c _(stmt)(s,N_(D)))  (2)

The number of processors has no influence in this case and the number ofparallel elements in the resulting plan is 1.

The cost can be quantified for each type of statement of the tuplecalculus discussed above. There can be two classes of statements: simplestatements such as assignments, expressions or calling the lookup or addfunction, conditional statements, and blocking statements for blockingstatements such as foreach or while. The cost of simple statements canbe constant (with the exception of the expression cost) and can be addedup to arrive at the final costs, while blocking statements themselves donot incur any costs but multiply the costs for all sub-statementsdepending on the number of iterations that are performed. Table 2illustrates the costs for all statements (first part—simple statements;second part—blocking statements).

TABLE 2 Costs of Statements. Statement Cost Description Assignmentc_(assign) Assignment cost Expression o * c_(expr) o is the number ofoperands in the expression Lookup( ) c_(lkp) cost of lookup functionc_(dict) initial cost of building dictionary, depends on N_(D) Add( )c_(add) cost of add function For each While$i*{\sum\limits_{s^{\prime}}^{S^{\prime}}\; {c_{stmt}\left( s^{\prime} \right)}}$S′ is the list of sub-statements of the loop statement and i is thenumber of iterations

As shown in Table 2, the costs of blocking statements depend on thenumber of loop iterations and costs of its sub-statements. As can beseen, the costs for the simple statements and blocking statements do notdirectly depend on the size of the input data. However, each script canstart with a loop over its input data, thus, the size N_(D) of the inputdata can determine the number of loop iterations i. The costs for allsub-statements of this loop can be multiplied with this number and thuscan indirectly depend on it.

To make a decision between a procedural script block and its declarativecounterpart, each declarative operator can have an assigned cost. Forexample, each declarative operator discussed above can have anassociated cost that can depend on the input data size and optionally asecond input parameter. Table 3 illustrates separate costs for eachoperator.

TABLE 3 Operator Costs. Operator Cost Selection c_(sel)(N_(D), sel) Joinc_(join)(N_(Left), N_(Right)) Projection c_(prj)(N_(D)) Aggregationc_(agg)(N_(D)) Assign c_(assgn)(N_(D)) Assignmerge c_(assgnmerge)(N_(D))

In contrast to the cost for simple statements that do not directlydepend on N_(D), each declarative operator can be modeled depending onits input size. For the join operator, this means that it depends on thesize of its left and right input. The selection operator can also have asecond parameter which can be the selectivity of its predicate.Furthermore, for some operators, such as aggregation, join, and/orprojection operators, there can exist some minimal cost for executingsuch operator even for a very small input size.

To calculate the cost for a hybrid plan, the costs for each declarativeoperator can be summed up together with the costs for all containedprocedural operators. For each procedural operator, the cost of thecontained script can be calculated by accumulating the cost of everybasic statement multiplied by the number of iterations of encompassinglooping statements. If the plan contains declarative and/or proceduraloperators that are independent of each other and can be calculated inparallel, the costs of the respective operators can be divided by thedegree of parallelism determined by the minimum of the number ofavailable processes and the number of parallel operators. Thus, theformula for the overall cost can be as follows, where Σ_(k) ^(N) ^(O)^(-N) ^(P) c_(decl) represents a sum of costs for all declarativestatements in the generated plan (whether translated or alreadyexisting), Σ_(m) ^(N) ^(p) c_(m) ^(proc) represents a sum of costs forall procedural statements in the generated plan; Σ^(c) ^(parop)/min(P_(c),P_(p)) represents a sum of costs for all operators(procedural and/or declarative) that are calculated in parallel asdivided by a minimum number of available parallel processors andparallel operators:

Cost=Σ_(k) ^(N) ^(O) ^(-N) ^(P) c _(decl)+Σ_(m) ^(Np) c _(m)^(proc)+Σ^(C) ^(parop) /min(P _(C) ,P _(p))  (3)

An experiment consistent with one or more implementations of the currentsubject matter illustrates some of the advantages of using a hybridquery execution plan and a cost model discussed above. Measurablequantities for different operator cost functions in the above-referencedcost model were obtained and predictions of the discussed cost modelagainst experimental results were validated. To do so, execution timewas measured for each declarative operator for different input sizes.Further, for a selection operator, selectivity of the selectionpredicate was varied and for a join operator, different input sizes forleft and right inputs were measured. During this experiment, operatorruntimes with a physical plan optimizer turned off were measured toensure that the physical execution plan closely resembles the logicalexecution plan. The measured operators were not optimized or combinedwith other operators.

Execution times of procedural patterns and their matching declarativecounterparts were compared for different input sizes N_(D). For smallerdata sets, the experiment indicated a better performance of theprocedural aggregation and projection operators only than theirdeclarative counterparts. For other operators and/or for larger datasets, declarative operators outperformed procedural operators.

Also, measurements of execution time of the script shown in FIG. 4indicated that use of declarative operators and/or a combination(hybrid) of declarative and procedural operators outperforms plans thatuse procedural operators, thereby validating the cost model. Hence, useof hybrid plans and/or plans that contain declarative operators reducesexecution costs.

FIG. 7 illustrates features of a method 700 consistent withimplementations of the current subject matter. At 702, a query executionplan describing a query for accessing data is received, for example at aprocessing system that includes one or more processors. The queryexecution plan includes a procedural pattern that, at 704 is matched toa stored pattern for which an equivalent declarative operator has beenpre-defined. At 706, a hybrid execution plan is generated by replacingthe procedural pattern with the equivalent declarative operator. At 710,a hybrid execution plan processing cost is assigned to execution of thehybrid execution plan and a query execution plan processing cost isassigned to execution of the query execution plan. The assigning caninclude evaluating a cost model for the hybrid execution plan and thequery execution plan. The query is executed at 712 using either thehybrid execution plan or the query execution plan according to which hasthe lower processing cost. The cost model can include a costcorresponding to processing time of the query execution plan. The costcan be determined based on the formula (3) discussed above.

Aspects of the subject matter described herein can be embodied insystems, apparatus, methods, and/or articles depending on the desiredconfiguration. In particular, various implementations of the subjectmatter described herein can be realized in digital electronic circuitry,integrated circuitry, specially designed application specific integratedcircuits (ASICs), computer hardware, firmware, software, and/orcombinations thereof. These various implementations can includeimplementation in one or more computer programs that are executableand/or interpretable on a programmable system including at least oneprogrammable processor, which can be special or general purpose, coupledto receive data and instructions from, and to transmit data andinstructions to, a storage system, at least one input device, and atleast one output device.

These computer programs, which can also be referred to programs,software, software applications, applications, components, or code,include machine instructions for a programmable processor, and can beimplemented in a high-level procedural and/or object-orientedprogramming language, and/or in assembly/machine language. As usedherein, the term “machine-readable medium” refers to any computerprogram product, apparatus and/or device, such as for example magneticdiscs, optical disks, memory, and Programmable Logic Devices (PLDs),used to provide machine instructions and/or data to a programmableprocessor, including a machine-readable medium that receives machineinstructions as a machine-readable signal. The term “machine-readablesignal” refers to any signal used to provide machine instructions and/ordata to a programmable processor. The machine-readable medium can storesuch machine instructions non-transitorily, such as for example as woulda non-transient solid state memory or a magnetic hard drive or anyequivalent storage medium. The machine-readable medium can alternativelyor additionally store such machine instructions in a transient manner,such as for example as would a processor cache or other random accessmemory associated with one or more physical processor cores.

To provide for interaction with a user, the subject matter describedherein can be implemented on a computer having a display device, such asfor example a cathode ray tube (CRT) or a liquid crystal display (LCD)monitor for displaying information to the user and a keyboard and apointing device, such as for example a mouse or a trackball, by whichthe user may provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well. For example,feedback provided to the user can be any form of sensory feedback, suchas for example visual feedback, auditory feedback, or tactile feedback;and input from the user may be received in any form, including, but notlimited to, acoustic, speech, or tactile input. Other possible inputdevices include, but are not limited to, touch screens or othertouch-sensitive devices such as single or multi-point resistive orcapacitive trackpads, voice recognition hardware and software, opticalscanners, optical pointers, digital image capture devices and associatedinterpretation software, and the like.

The subject matter described herein can be implemented in a computingsystem that includes a back-end component, such as for example one ormore data servers, or that includes a middleware component, such as forexample one or more application servers, or that includes a front-endcomponent, such as for example one or more client computers having agraphical user interface or a Web browser through which a user caninteract with an implementation of the subject matter described herein,or any combination of such back-end, middleware, or front-endcomponents. A client and server are generally, but not exclusively,remote from each other and typically interact through a communicationnetwork, although the components of the system can be interconnected byany form or medium of digital data communication. Examples ofcommunication networks include, but are not limited to, a local areanetwork (“LAN”), a wide area network (“WAN”), and the Internet. Therelationship of client and server arises by virtue of computer programsrunning on the respective computers and having a client-serverrelationship to each other.

The implementations set forth in the foregoing description do notrepresent all implementations consistent with the subject matterdescribed herein. Instead, they are merely some examples consistent withaspects related to the described subject matter. Although a fewvariations have been described in detail herein, other modifications oradditions are possible. In particular, further features and/orvariations can be provided in addition to those set forth herein. Forexample, the implementations described above can be directed to variouscombinations and sub-combinations of the disclosed features and/orcombinations and sub-combinations of one or more features further tothose disclosed herein. In addition, the logic flows depicted in theaccompanying figures and/or described herein do not necessarily requirethe particular order shown, or sequential order, to achieve desirableresults. The scope of the following claims may include otherimplementations or embodiments.

1-20. (canceled)
 21. A computer implemented method comprising:translating a first procedural construct in a received query plandescribing a query for accessing data into a first declarative operatorthat is equivalent to the first procedural construct; generating a costmodel based on at least the first declarative operator; determining,using the generated cost model, an effectiveness of a hybrid queryexecution plan and the query execution plan, the effectiveness of thehybrid query execution plan associated with a processing cost of thehybrid query execution plan and the effectiveness of the query executionplan associated with a processing cost of the query execution plan, thehybrid query execution plan including a combination of the at least oneother procedural construct of the query execution plan and the firstdeclarative operator; and executing the query using the hybrid executionplan when the determined effectiveness of the hybrid execution plan isgreater than the determined effectiveness of the query execution plan.22. The computer implemented method as in claim 21, wherein the firstequivalent declarative operator comprises a conversion of procedurallogic within the first procedural construct, the conversion comprisingimplicitly unrolling loops within the procedural logic and independentlycalculating an expression for each tuple within the procedural logic.23. The computer implemented method as in claim 21, further comprisingapplying tuple calculus to identify a procedural statement forreplacement of the first procedural statement by the first declarativestatement.
 24. The computer implemented method as in claim 21, whereinthe cost model is configured to determine the hybrid execution planprocessing cost and the query execution plan processing cost usingfunctions comprising:${Cost} = {{\sum_{K}^{N_{o} - N_{p}}C_{decl}} + {\sum_{m}^{N_{p}}C_{m}^{proc}} + \frac{\sum^{C_{prop}}}{\min \left( {P_{C},P_{P}} \right)}}$where Σ_(K) ^(N) ^(o) ^(-N) ^(p) C_(decl) represents a first sum ofcosts for all declarative statements, Σ_(m) ^(N) ^(p) C_(m) ^(proc)represents a second sum of costs for all procedural statements, and$\frac{\sum^{C_{prop}}}{\min \left( {P_{c},P_{P}} \right)}$ representsa third sum of costs for all procedural and/or declarative operatorsthat are calculated in parallel as divided by a minimum number ofavailable parallel processors (P_(C)) and parallel operators (P_(P)).25. The computer implemented method as in claim 21, wherein the firstdeclarative operator comprises at least one of a selection declarativeoperator, a join declarative operator, a leftouter-join declarativeoperator, a projection declarative operator, an aggregation declarativeoperator, and an assign declarative operator.
 26. The computerimplemented method as in claim 21, further comprising: translating asecond procedural construct in the query plan to a second declarativeoperator, equivalent to the second procedural construct, in the hybridexecution plan, the translating comprising identifying a borderlineprocedural construct associated with a side effect when the borderlineprocedural construct is executed.
 27. The computer implemented method asin claim 26 wherein the side effect enables a condition that when afirst tuple element value is changed during a loop iteration of theborderline procedural construct, the changed tuple element value isaccessed in a subsequent loop iteration such that access of the changedtuple element value rather than access of the first tuple element valueprevents the borderline procedural construct to be translated to anequivalent declarative operator.
 28. The computer program product as inclaim 21, further comprising: generating a second hybrid execution planby replacing a second procedural construct with a second declarativeoperator equivalent to the second procedural construct; determining,using the cost model, an effectiveness of the second hybrid queryexecution plan, the effectiveness of the second hybrid query executionplan associated with a processing cost for the second hybrid queryexecution plan; and executing the query using the second hybridexecution plan if the effectiveness of the second hybrid execution planis greater than the effectiveness of the hybrid query execution plan.29. A system comprising: at least one programmable processor; and atleast one machine-readable medium storing instructions that, whenexecuted by the at least one programmable processor, cause the at leastone programmable processor to perform operations comprising: receiving aquery execution plan describing a query for accessing data andcomprising a first procedural construct; translating the firstprocedural construct into a first declarative operator equivalent, thefirst declarative operator equivalent to the first procedural construct;generating a cost model based on at least the first declarativeoperator; determining, using the generated cost model, an effectivenessof a hybrid query execution plan and the query execution plan, theeffectiveness of the hybrid query execution plan associated with aprocessing cost of the hybrid query execution plan and the effectivenessof the query execution plan associated with a processing cost of thequery execution plan, the hybrid query execution plan including acombination of the at least one other procedural construct of the queryexecution plan and the first declarative operator; executing the queryusing the hybrid execution plan after determining that the effectivenessof the hybrid execution plan is greater than the effectiveness of thequery execution plan.
 30. The system as in claim 29, wherein the firstequivalent declarative operator comprises a conversion of procedurallogic within the first procedural construct, the conversion comprisingimplicitly unrolling loops within the procedural logic and independentlycalculating an expression for each tuple within the procedural logic.31. The system as in claim 29, wherein the operations further comprise:applying tuple calculus to identify a procedural statement forreplacement of the first procedural statement by the first declarativestatement.
 32. The system as in claim 29, wherein the cost model isconfigured to determine the hybrid execution plan processing cost andthe query execution plan processing cost using functions comprising:${Cost} = {{\sum_{K}^{N_{o} - N_{p}}C_{decl}} + {\sum_{m}^{N_{p}}C_{m}^{proc}} + \frac{\sum^{C_{prop}}}{\min \left( {P_{C},P_{P}} \right)}}$where Σ_(K) ^(N) ^(o) ^(-N) ^(p) C_(decl) represents a first sum ofcosts for all declarative statements, Σ_(m) ^(N) ^(p) C_(m) ^(proc)represents a second sum of costs for all procedural statements, and$\frac{\sum^{C_{prop}}}{\min \left( {P_{c},P_{P}} \right)}$ representsa third sum of costs for all procedural and/or declarative operatorsthat are calculated in parallel as divided by a minimum number ofavailable parallel processors (P_(C)) and parallel operators (P_(P)).33. The system as in claim 29, wherein the first declarative operatorcomprises at least one of a selection declarative operator, a joindeclarative operator, a leftouter-join declarative operator, aprojection declarative operator, an aggregation declarative operator,and an assign declarative operator.
 34. The system as in claim 29,wherein the operations further comprise: translating a second proceduralconstruct in the query plan to a second declarative operator, equivalentto the second procedural construct, in the hybrid execution plan, thetranslating comprising identifying a borderline procedural constructassociated with a side effect when the borderline procedural constructis executed.
 35. The system as in claim 34, wherein the side effectenables a condition that when a first tuple element value is changedduring a loop iteration of the borderline procedural construct, thechanged tuple element value is accessed in a subsequent loop iterationsuch that access of the changed tuple element value rather than accessof the first tuple element value prevents the borderline proceduralconstruct to be translated to an equivalent declarative operator. 36.The system as in claim 29, wherein the operations further comprise:generating a second hybrid execution plan by replacing a secondprocedural construct with a second declarative operator equivalent tothe second procedural construct; determining, using the cost model, aneffectiveness of the second hybrid query execution plan, theeffectiveness of the second hybrid query execution plan associated witha processing cost for the second hybrid query execution plan; andexecuting the query using the second hybrid execution plan if theeffectiveness of the second hybrid execution plan is greater than theeffectiveness of the hybrid query execution plan.
 37. A computer programproduct comprising a machine-readable medium storing instructions that,when executed by the at least one programmable processor, cause the atleast one programmable processor to perform operations comprising:translating a first procedural construct in a received query plandescribing a query for accessing data into a first declarative operatorthat is equivalent to the first procedural construct; generating a costmodel based on at least the first declarative operator; determining,using the generated cost model, an effectiveness of a hybrid queryexecution plan and the query execution plan, the effectiveness of thehybrid query execution plan associated with a processing cost of thehybrid query execution plan and the effectiveness of the query executionplan associated with a processing cost of the query execution plan, thehybrid query execution plan including a combination of the at least oneother procedural construct of the query execution plan and the firstdeclarative operator; and, executing the query using the hybridexecution plan when the determined effectiveness of the hybrid executionplan is greater than the determined effectiveness of the query executionplan.
 38. The computer program product as in claim 37, wherein the firstequivalent declarative operator comprises a conversion of procedurallogic within the first procedural construct, the conversion comprisingimplicitly unrolling loops within the procedural logic and independentlycalculating an expression for each tuple within the procedural logic.39. The computer program product as in claim 37, wherein the operationsfurther comprise applying tuple calculus to identify a proceduralstatement for replacement of the first procedural statement by the firstdeclarative statement.
 40. The computer program product as in claim 37,wherein the cost model is configured to determine the hybrid executionplan processing cost and the query execution plan processing cost usingfunctions comprising:${Cost} = {{\sum_{K}^{N_{o} - N_{p}}C_{decl}} + {\sum_{m}^{N_{p}}C_{m}^{proc}} + \frac{\sum^{C_{prop}}}{\min \left( {P_{C},P_{P}} \right)}}$where Σ_(K) ^(N) ^(o) ^(-N) ^(p) C_(decl) represents a first sum ofcosts for all declarative statements, Σ_(m) ^(N) ^(p) C_(m) ^(proc)represents a second sum of costs for all procedural statements, and$\frac{\sum^{C_{prop}}}{\min \left( {P_{c},P_{P}} \right)}$ representsa third sum of costs for all procedural and/or declarative operatorsthat are calculated in parallel as divided by a minimum number ofavailable parallel processors (P_(C)) and parallel operators (P_(P)).41. The computer program product as in claim 37, wherein the operationsfurther comprise: translating a second procedural construct in the queryplan to a second declarative operator, equivalent to the secondprocedural construct, in the hybrid execution plan, the translatingcomprising identifying a borderline procedural construct associated witha side effect when the borderline procedural construct is executed. 42.The computer program product as in claim 37, wherein the operationsfurther comprise: generating a second hybrid execution plan by replacinga second procedural construct with a second declarative operatorequivalent to the second procedural construct; determining, using thecost model, an effectiveness of the second hybrid query execution plan,the effectiveness of the second hybrid query execution plan associatedwith a processing cost for the second hybrid query execution plan; andexecuting the query using the second hybrid execution plan if theeffectiveness of the second hybrid execution plan is greater than theeffectiveness of the hybrid query execution plan.